> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/mlfoundations/open_clip/llms.txt
> Use this file to discover all available pages before exploring further.

# Remote Training

> Train models with automatic cloud storage backup and resume from remote checkpoints

OpenCLIP supports remote training workflows that allow you to resume from remote checkpoints and automatically back up training progress to cloud storage. This is particularly useful for large-scale training on cloud infrastructure or when using shared storage systems.

## Overview

Remote training features in OpenCLIP:

1. **Resume from remote paths**: Load checkpoints directly from S3 or other remote storage
2. **Automatic backup**: Continuously sync training checkpoints to remote storage
3. **fsspec support**: Work with any filesystem supported by fsspec
4. **Checkpoint management**: Automatically clean up old checkpoints to save space

## Resuming from Remote Checkpoints

You can resume training directly from a remote checkpoint without downloading it first.

### Resume from S3

Use the S3 URI directly in the `--resume` flag:

```bash theme={null}
python -m open_clip_train.main \
    --model ViT-B-32 \
    --resume s3://my-bucket/checkpoints/epoch_10.pt \
    --train-data "/data/train.tar" \
    --dataset-type webdataset \
    --batch-size 256 \
    --epochs 32
```

### Resume from Other Remote Storage

OpenCLIP uses `fsspec` to support various storage backends:

```bash theme={null}
# Google Cloud Storage
--resume gs://my-bucket/checkpoints/epoch_10.pt

# Azure Blob Storage
--resume az://my-container/checkpoints/epoch_10.pt

# HTTP/HTTPS
--resume https://my-server.com/checkpoints/epoch_10.pt
```

### Complete Resume Example

```bash theme={null}
python -m open_clip_train.main \
    --train-data "/data/laion400m/train-{0000..4000}.tar" \
    --train-num-samples 400000000 \
    --dataset-type webdataset \
    --batch-size 256 \
    --precision amp \
    --workers 8 \
    --model ViT-L-14 \
    --resume s3://my-training-bucket/runs/vitl14-run1/checkpoints/epoch_15.pt \
    --epochs 32 \
    --lr 1e-3 \
    --warmup 2000
```

## Automatic Remote Backup

Continuously back up training checkpoints to remote storage during training. This prevents data loss and enables easy resume from any point.

### Basic Remote Sync Setup

Use `--remote-sync` to specify the remote destination:

```bash theme={null}
python -m open_clip_train.main \
    --model ViT-B-32 \
    --train-data "/data/train.tar" \
    --dataset-type webdataset \
    --batch-size 256 \
    --epochs 32 \
    --logs /scratch/training \
    --remote-sync s3://my-bucket/training-runs \
    --name my-experiment
```

This will:

1. Save checkpoints locally to `/scratch/training/my-experiment/`
2. Sync to `s3://my-bucket/training-runs/my-experiment/`
3. Run sync in background every 5 minutes (default)

### Remote Sync Parameters

#### --remote-sync

Specify the remote path for backup:

```bash theme={null}
--remote-sync s3://my-bucket/training-runs
```

**Supported formats:**

* S3: `s3://bucket-name/path`
* S3 with credentials: `s3://bucket-name/path` (uses AWS credentials)
* Other fsspec backends: `gs://`, `az://`, etc.

#### --remote-sync-frequency

How often to sync (in seconds):

```bash theme={null}
--remote-sync-frequency 300  # Sync every 5 minutes (default)
--remote-sync-frequency 600  # Sync every 10 minutes
--remote-sync-frequency 1800 # Sync every 30 minutes
```

**Recommendations:**

* Fast storage: 300 seconds (5 minutes)
* Slow storage: 900-1800 seconds (15-30 minutes)
* Large checkpoints: 600+ seconds
* Small checkpoints: 300 seconds

#### --remote-sync-protocol

Specify the sync protocol:

```bash theme={null}
--remote-sync-protocol s3      # Use S3 protocol (default, recommended)
--remote-sync-protocol fsspec  # Use fsspec (experimental, slow)
```

**Note:** The `fsspec` protocol is currently experimental and very slow. Use `s3` for production workloads.

## Complete Remote Training Examples

### Example 1: S3 Training with Backup

Train with local SSD and automatic S3 backup:

```bash theme={null}
python -m open_clip_train.main \
    --save-frequency 1 \
    --zeroshot-frequency 1 \
    --report-to tensorboard \
    --train-data "/data/cc12m/train-{0000..2175}.tar" \
    --train-num-samples 10968539 \
    --dataset-type webdataset \
    --batch-size 320 \
    --precision amp \
    --workers 8 \
    --model ViT-B-32 \
    --warmup 2000 \
    --lr 1e-3 \
    --wd 0.1 \
    --epochs 32 \
    --logs /scratch/openclip \
    --remote-sync s3://my-training-bucket/experiments \
    --remote-sync-frequency 300 \
    --name vitb32-cc12m \
    --imagenet-val /data/imagenet/validation/
```

The sync process:

1. Local checkpoints: `/scratch/openclip/vitb32-cc12m/checkpoints/`
2. Remote backup: `s3://my-training-bucket/experiments/vitb32-cc12m/checkpoints/`
3. Syncs every 5 minutes
4. Final sync when training completes

### Example 2: Multi-GPU with Remote Sync

```bash theme={null}
torchrun --nproc_per_node 8 -m open_clip_train.main \
    --train-data "/data/laion400m/train-{0000..4000}.tar" \
    --train-num-samples 400000000 \
    --dataset-type webdataset \
    --batch-size 256 \
    --precision amp \
    --workers 8 \
    --model ViT-L-14 \
    --grad-checkpointing \
    --local-loss \
    --gather-with-grad \
    --warmup 2000 \
    --lr 1e-3 \
    --epochs 32 \
    --logs /scratch/training \
    --remote-sync s3://my-bucket/large-runs \
    --remote-sync-frequency 600 \
    --delete-previous-checkpoint \
    --name vitl14-laion400m
```

### Example 3: Resume from S3 and Continue Syncing

```bash theme={null}
python -m open_clip_train.main \
    --model ViT-B-32 \
    --resume s3://my-bucket/experiments/run1/checkpoints/epoch_10.pt \
    --train-data "/data/train.tar" \
    --dataset-type webdataset \
    --batch-size 256 \
    --epochs 32 \
    --logs /scratch/training \
    --remote-sync s3://my-bucket/experiments \
    --name run1
```

## Checkpoint Management

### Delete Previous Checkpoints

Save disk space by automatically deleting old checkpoints:

```bash theme={null}
--delete-previous-checkpoint
```

This will:

* Keep only the most recent checkpoint locally
* Delete previous checkpoints after saving a new one
* Remote backups are unaffected (all checkpoints synced)
* Useful when local storage is limited

**Example:**

```bash theme={null}
python -m open_clip_train.main \
    --model ViT-L-14 \
    --train-data "/data/train.tar" \
    --dataset-type webdataset \
    --batch-size 256 \
    --epochs 32 \
    --logs /scratch/training \
    --remote-sync s3://my-bucket/runs \
    --delete-previous-checkpoint \
    --name vitl14-experiment
```

**Result:**

* Local: Only `epoch_latest.pt` is kept
* Remote: All epochs synced to S3 (`epoch_1.pt`, `epoch_2.pt`, etc.)

### Resume Latest from Remote

When using `--resume latest` with remote sync:

```bash theme={null}
python -m open_clip_train.main \
    --model ViT-B-32 \
    --resume latest \
    --train-data "/data/train.tar" \
    --dataset-type webdataset \
    --batch-size 256 \
    --epochs 32 \
    --logs /scratch/training \
    --remote-sync s3://my-bucket/runs \
    --remote-sync-protocol s3 \
    --name my-experiment
```

**Important limitations:**

* Only works with `--remote-sync-protocol s3`
* Does not work with `--save-most-recent`
* Checks remote storage for latest checkpoint
* May not find checkpoint if sync is still in progress

## AWS S3 Configuration

### AWS Credentials

Ensure AWS credentials are configured:

```bash theme={null}
# Option 1: AWS CLI configuration
aws configure

# Option 2: Environment variables
export AWS_ACCESS_KEY_ID=your_access_key
export AWS_SECRET_ACCESS_KEY=your_secret_key
export AWS_DEFAULT_REGION=us-west-2

# Option 3: IAM role (on EC2)
# Credentials automatically available
```

### S3 Bucket Setup

```bash theme={null}
# Create bucket
aws s3 mb s3://my-training-bucket

# Set lifecycle policy to manage old checkpoints
aws s3api put-bucket-lifecycle-configuration \
    --bucket my-training-bucket \
    --lifecycle-configuration file://lifecycle.json
```

**lifecycle.json:**

```json theme={null}
{
  "Rules": [
    {
      "Id": "DeleteOldCheckpoints",
      "Status": "Enabled",
      "Prefix": "experiments/",
      "Expiration": {
        "Days": 90
      }
    }
  ]
}
```

## Workflow Patterns

### Pattern 1: Fast Local Storage + S3 Backup

Best for: Training on cloud instances with local SSDs

```bash theme={null}
# Use local SSD for speed
--logs /scratch/training

# Back up to S3 for durability
--remote-sync s3://my-bucket/runs

# Clean up local storage
--delete-previous-checkpoint
```

### Pattern 2: Resume After Interruption

Best for: Spot instances, preemptible VMs

```bash theme={null}
# Initial training
python -m open_clip_train.main \
    --model ViT-B-32 \
    --logs /scratch/training \
    --remote-sync s3://my-bucket/runs \
    --name experiment-1 \
    ...

# After interruption, resume
python -m open_clip_train.main \
    --model ViT-B-32 \
    --resume s3://my-bucket/runs/experiment-1/checkpoints/epoch_latest.pt \
    --logs /scratch/training \
    --remote-sync s3://my-bucket/runs \
    --name experiment-1 \
    ...
```

### Pattern 3: Centralized Checkpoint Storage

Best for: Team collaboration, multiple training nodes

```bash theme={null}
# All nodes sync to same bucket
--remote-sync s3://team-bucket/shared-experiments

# Each experiment has unique name
--name ${USER}-${MODEL}-${DATE}

# Team members can resume from any checkpoint
--resume s3://team-bucket/shared-experiments/alice-vitb32-2024/checkpoints/epoch_10.pt
```

## Sync Process Details

### How Sync Works

1. Training starts, background sync process launched
2. Every `--remote-sync-frequency` seconds:
   * Sync all files from local logs directory to remote
   * Only uploads changed/new files
   * Sync happens in background, doesn't block training
3. When training completes:
   * Final sync ensures all checkpoints uploaded
   * Process waits for final sync to complete

### What Gets Synced

All files in the local logs directory:

* Checkpoints (`epoch_*.pt`)
* TensorBoard logs
* Training logs
* Configuration files
* Any other files in the directory

### Sync Command (S3)

Under the hood, uses AWS CLI:

```bash theme={null}
aws s3 sync /local/path s3://bucket/remote/path --exact-timestamps
```

## Performance Considerations

### Sync Frequency

**Too frequent:**

* Wastes bandwidth
* May impact training performance
* Unnecessary for large checkpoints

**Too infrequent:**

* Risk losing more progress on failure
* Longer wait for final sync

**Recommended:**

* Small models: 300-600 seconds
* Large models: 600-1800 seconds
* Fast networks: 300 seconds
* Slow networks: 900+ seconds

### Network Impact

* Sync runs in background process
* Minimal impact on training throughput
* May affect data loading if sharing bandwidth
* Use local data loading when possible

## Troubleshooting

### Sync Failing

**Check AWS credentials:**

```bash theme={null}
aws s3 ls s3://my-bucket/
```

**Check bucket permissions:**

```bash theme={null}
aws s3api get-bucket-acl --bucket my-bucket
```

**Test manual sync:**

```bash theme={null}
aws s3 sync /local/path s3://my-bucket/test-path
```

### Resume from S3 Failing

**Verify checkpoint exists:**

```bash theme={null}
aws s3 ls s3://my-bucket/path/to/checkpoint.pt
```

**Check file size:**

```bash theme={null}
aws s3 ls s3://my-bucket/path/to/checkpoint.pt --human-readable
```

**Test download manually:**

```bash theme={null}
aws s3 cp s3://my-bucket/path/to/checkpoint.pt /tmp/test.pt
```

### Slow Sync

* Use `--remote-sync-protocol s3` (not `fsspec`)
* Increase `--remote-sync-frequency`
* Check network bandwidth
* Consider using S3 Transfer Acceleration
* Reduce checkpoint size if possible

### "Resume latest" Not Finding Checkpoint

* Ensure sync completed before trying to resume
* Check remote path matches expectations
* Use explicit checkpoint path instead of "latest"
* Verify `--remote-sync-protocol s3` is set

## Best Practices

1. **Always use remote sync for long training runs**
   * Prevents data loss from hardware failures
   * Enables easy resume from any point

2. **Use local fast storage with remote backup**
   * Local SSD for training speed
   * S3 for durability and sharing

3. **Delete old local checkpoints**
   * Use `--delete-previous-checkpoint`
   * Keep all checkpoints in remote storage

4. **Set appropriate sync frequency**
   * Balance between safety and performance
   * Consider checkpoint size and network speed

5. **Test sync before long runs**
   * Verify credentials and permissions
   * Test manual sync first
   * Monitor first few syncs

6. **Use unique experiment names**
   * Prevents conflicts in shared storage
   * Makes checkpoints easy to find
   * Include timestamp or identifier

7. **Monitor sync process**
   * Check logs for sync errors
   * Verify files appearing in remote storage
   * Test resume before needing it